Dynamic provisioning of a virtual test environment

ABSTRACT

At least one test of a first computing system is launched utilizing a second computing system that includes a first set of computing devices. Progress of the test is monitored during a first period of time. Performance of the second computing system is also monitored during the first period. An additional second set of computing devices is automatically provisioned for inclusion in the second computing system based at least in part on the monitoring of the test progress and monitoring of the performance of the computing system during the first time period. The test can utilize the first and second sets of computing devices during a second period of time subsequent to the first period.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation (and claims the benefit of priorityunder 35 U.S.C. §120) of U.S. application Ser. No. 13/155,371, filedJun. 7, 2011 and entitled DYNAMIC PROVISIONING OF A VIRTUAL TESTENVIRONMENT. The disclosure of the prior application is considered partof and is incorporated by reference in the disclosure of thisapplication.

TECHNICAL FIELD

This disclosure relates in general to the field of computer systemstesting and, more particularly virtual test environments.

BACKGROUND

Virtualization and cloud computing have been increasing in popularity.Hardware virtualization has been used, for instance, to run multiplecopies of an operating system as virtual machines (VMs) within a single,physical hardware device. Hardware virtualization can offer cost,flexibility and risk management benefits for the applications utilizingthe virtualized hardware. Virtual endpoints have been used, for instancein service-oriented architectures (SOA), allow the SOA to define virtuallocations for services that need to be invoked, while shielding theactual end point of the service itself. This can be used, for instance,to allow for the physical address (or URL) of a service to be changed,depending upon when and how the service is used as part of a givenworkflow. Virtualized services themselves have also been developed(e.g., in iTKO Corporation's LISA™ testing suite) and described, such asin U.S. patent application Ser. No. 12/242,783 to Michelsen (filed Sep.30, 2008). Virtual services can be constructed synthetically from WSDL,or modeled from existing services and underlying implementations, andcan be used to streamline testing, development, and deployment practicesas a whole.

SUMMARY

This specification describes technologies relating to automaticprovisioning of a test server system. In general, one aspect of thesubject matter described in this specification can be embodied inmethods that include the actions of launching at least one test of afirst computing system utilizing a second computing system including afirst set of computing devices and monitoring progress of the testduring a first period of time. Performance of the second computingsystem can also be monitored during the first period. An additionalsecond set of computing devices can be automatically provisioned forinclusion in the second computing system based at least in part on themonitoring of the test progress and monitoring of the performance of thecomputing system during the first time period. The test can utilize thefirst and second sets of computing devices during a second period oftime subsequent to the first period.

These and other embodiments can each optionally include one or more ofthe following features. The first and second sets of computing devicescan be remote from the first computing system. The second computingsystem can execute at least one instance of a virtual test labconfigured to simulate a set of interactions with the first computingsystem, and the virtual test lab can be executed on at least one of thefirst set of computing devices. Automatically provisioning the secondset of computing devices can include replicating instances of thevirtual test lab on at least some of the second set of computingdevices. Each instance of the virtual test lab can be executed using atleast one first virtual machine and a virtual instance of the firstcomputing system is tested, the virtual instance of the first computingsystem executed using at least one second virtual machine. Monitoringthe test during the first time period can include predictingrequirements of the test in the second period of time and the second setof computing devices can be provisioned based, at least in part, on thepredicted requirements of the test. Monitoring performance of the secondcomputing system can include monitoring computing capacity of the firstset of computing devices. Computing capacity can relate to at least oneof processing capacity of computing devices in the second computingsystem, available network bandwidth within the second computing system,and available memory of computing devices in the second computingsystem. The first and second sets of computing devices can beprovisioned from a plurality of cloud-based server devices. Performanceof the second computing system can be monitored during the secondperiod, the second computing system including the first and second setsof computing devices. It can be determined, based at least in part onmonitoring performance of the second computing system during the secondperiod, whether additional computing devices should be added to thesecond computing system in a third period subsequent to the secondperiod. It can be determined, based on the monitoring of the secondcomputing system during the second period, that at least a portion ofthe computing devices provisioned to the second computing system be atleast temporarily de-allocated from the second computing system.

Further, embodiments can each optionally include one or more of thefollowing features. At least one particular computing device, remotefrom the first computing system and within the second computing system,can execute at least one test coordinator engine managing the test. Theat least one test coordinator engine can monitor the test, monitorperformance of the second computing system, and initiate dynamicprovisioning of additional computing devices for inclusion within thesecond computing system during the test. User-defined parameters can beidentified for the test, where dynamic provisioning of additionalcomputing devices for inclusion within the second computing systemduring the test is based at least in part on the parameters. Parameterscan include a monitoring rate defining when performance of the secondcomputing system should be monitored in connection with a decisions toinitiate dynamic provisioning of addition computing devices forinclusion within the second computing system during the test, an initialsystem size for the second computing system, and/or a maximum systemsize for the second computing system, wherein the second computingsystem will not be automatically provisioned with additional computingdevices in excess of the maximum system size. The at least one test caninclude a plurality of tests, the plurality of tests including a firsttest and a second test executed at least partially in parallel. Eachtest in the plurality of tests can have a corresponding test coordinatorengine executed on the second computing system. Monitoring the at leastone test can include determining whether additional computing devicesshould be automatically provisioned to accommodate execution of testcoordinator engines for at least the first and second tests.User-defined parameter can be identified including instructions forcompleting each of the plurality of tests within a particular timeperiod, the particular time period including at least the first andsecond periods of time. Performance the second computing system duringthe first period can be monitored to determine whether additionalcomputing devices should be automatically provisioned for inclusion inthe second computing system in order to the plurality of tests withinthe particular time period. Performance of the second computing systemcan be monitored substantially continuously during the test anddecisions to automatically provision the second computing system withadditional computing devices are made substantially periodically.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. For instance, dynamic provisioning of a test labcan increase efficiency in setting-up and executing a test of acomputing system, among other advantages. For example, rather thanspending time and potentially wasting human and computing resourcesdetermining and allocating a sufficient or optimized amount of testinghardware for a test, test servers can be automatically provisioned, fromthe cloud in substantially real time while a test is being run, in orderto identify a sufficient or optimized amount of test hardware.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example system including a virtual testenvironment.

FIG. 2 is a schematic representation of an virtual test environment.

FIG. 3 is a flowchart of an example technique for dynamically allocatingcomputing devices for use in a virtual test environment.

FIGS. 4A-4E illustrate representations of example dynamic allocating ofcomputing devices for use in a virtual test environment.

FIGS. 5A-5D illustrate representations of additional example dynamicallocating of computing devices for use in a virtual test environment.

FIG. 6 illustrates a status graph of an example software load test usinga virtual test environment.

FIGS. 7A-7C illustrate example screenshots of UIs used in connectionwith a virtual test manager.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Virtual test environments, or labs, can be implemented using a pluralityof computing devices, including computing devices within a cloud-basedcomputing environment or network. In typical software tests,considerable time can be devoted to determining the hardwarerequirements of the very system responsible for performing the test.Properly provisioning the test lab with the proper hardware and softwareresources can be critical, as development decisions involving the systemunder test can be based on the results of the test. Accordingly, if atest lab is provisioned incorrectly or inadequately, the accuracy andefficacy of test results generated by the test can be unreliable orimprecise. For instance, if a test lab is implemented using a hardwaresystem that cannot perform at a level required for the test, the testcan fail or manifest as a bottleneck in interactions between the testingsystem and the system under test. Traditionally, fear of inadequatelyprovisioning the hardware for a test environment has resulted in lengthyanalyses and/or over-provisioning of hardware resources to ensure thatthere is enough hardware available to implement the test. Indeed, insome instances, the test system is itself tested to ensure that it iscorrectly provisioned. Such approaches, however, can be expensive andinefficient. For instance, lengthy and expensive simulations and testingof the testing system can be conducted simply to determine a minimumnumber of servers needed to perform a test. In many instances, testingadministrators elect to purchase a quantity of test hardware well inexcess of what will likely be needed, in order to ensure that enoughtest hardware is available. Over-provisioning can be particularlyinefficient, however, given the time and human resources required toprovision the (over-) allocated servers with the test simulation andmanagement software specific to the test, as well as repurposing theservers once the test is completed, using traditional techniques.

In improved implementations, the hardware implementing the test can bemonitored while the test is running to ensure that the testing hardwareis able to handle the loads introduced through execution of the test.Statistical data can be collected during the test to forecast trendsboth within the test's progress as well as with the testing hardware.Statistical data and trend calculations can then be used to determinewhether the set of hardware originally-provisioned for the test isadequate to accurately complete the test. Based on this assessment,additional hardware can be quickly and dynamically (e.g., automatically)allocated and provisioned, in real time, from a pool of availablehardware resources, such as a pool of available cloud servers, toaddress the forecasted need. In other instances, forecasted trends canindicate that too much hardware has been provisioned for a given test,allowing the test environment to dynamically release and tear downexcess servers for use by other users, systems, or, indeed, other tests.

Turning to the example implementation of FIG. 1, the illustratedcomputing system 100 includes, or is communicably coupled with, one ormore clients 102, 104, one or more application (or applicationdevelopment) servers (e.g., 106, 108), such as application serverswithin an enterprise software environment, for instance, using one ormore networks 120. The system 100 can further include a virtual testenvironment 110 adapted to implement one or more tests on computingsystems and applications, such as systems and applications (e.g., 112,114) hosted, developed, or otherwise available through applicationservers 106, 108, including other systems upon which the applicationsdepend, such as an SOA, composite software system, enterprise softwaresystem, or other system or subsystem. The virtual test environment 110can be implemented, at least in part, on a plurality of computingdevices, such as servers (e.g., servers 115-122) in a server farm,server pool, cloud system, etc. Portions of the virtual test environmentcan also be executed persistently, for instance, on clients 102, 104and/or servers 106, 108. Servers 115-122 in virtual test environment 110can be dedicated testing servers, or general purpose computing devices,such as cloud servers available on demand through one or more cloudcomputing services.

In general, each of “servers” 106, 108, 115-122 can comprise electroniccomputing devices operable to receive, transmit, process, store, ormanage data and information associated with the software system 100. Asused in this document, the term “computer” or “computing device” isintended to encompass any suitable processing device. For example, thesystem 100 may be implemented using computers other than servers,including server pools. Further, any, all, or some of the servers 106,108, 115-122 may be adapted to execute any operating system, includingLinux, UNIX, Windows Server, etc., as well as virtual machines (e.g.,125-132) adapted to virtualize execution of a particular operatingsystem, including customized and proprietary operating systems.

Servers 115-122 and other computing devices (e.g., 115-122) in virtualtesting environment 110 can each include one or more processors (e.g.,130-137), computer-readable memory (e.g., 140-147), and one or moreinterfaces. Each server can further include code that creates anexecution environment for the computer program in question, such as oneor more virtual machines (“VMs”) (e.g., 125-132). Additional softwaremodules, applications, simulators, and functionality can also beexecuted on servers 115-122, including within virtual machines 125-132.In some instances, a virtual test lab corresponding to a particular testof a particular computing system, application, or software component canbe implemented by using one or more computing devices in virtual testingenvironment 110. For instance, in one example, a system under test canbe virtualized as a system simulator 150 executed on a virtual machine125 on one or more servers (e.g., 115) in the virtual testingenvironment 110. One or more other servers (e.g., 116) can be used toexecute a test coordinator 155 adapted to monitor and manage the test onthe system modeled in system simulator 150. Additionally, one or moreinstances of a test simulator (e.g., 158 a-c) can be provisioned on oneor more servers (e.g., 117-119) of virtual testing environment 110, eachtest simulator adapted to perform or model an interaction or transactionwith the system under test (e.g., a virtual system simulator 150) andperform operations and simulations in connection with the particulartest. Test coordinator 155 can monitor performance of the test, as wellas performance of test simulators 158 a-c and computing devices (e.g.,117-119) used to implement the simulators 158 a-c to ensure the testproceeds as it should, including initiating and directing theprovisioning of additional servers (e.g., 120-122), if necessary, tocomplete the particular test, or a plurality, or suite, of tests. Insome implementations, persistent test environment modules, such as atest manager 170 can also be provided, to assist with the monitoring andmanagement of tests and allocated resources (such as servers 115-119),including the dynamic provisioning of additional test servers (e.g.,servers 120-122).

In some instances, a particular system can be tested using virtualtesting environment 110 by virtualizing the system and components undertest into virtual services, allowing the test to be run on a dedicated,virtualized clone (or simulator 150) of the system under test, ratherthan the live system itself. Virtualizing the system under test,including the system's dependencies, as well as the can be advantageous,so as to more fully leverage the extensible capacity of a testingenvironment utilizing an on-demand computing architecture, such as acloud service. As an example, a particular system under test, such as anorder management system, may require access to a mainframe. It may notbe desirable, or feasible, however, to provision the entire mainframe onthe cloud. Unfortunately, if the mainframe is a constrained systemand/or off-cloud, virtualizing other system components on the cloudinfrastructure may do little to increase the capacity and efficiency ofthe overall test environment. Indeed, virtualizing and provisioningadditional capacity in the front-end architecture under test can behandicapped by backend capacity issues that cannot or may not beavailable for virtualization.

In some instances, the behavior of large, constrained, third-party, orother “out-of-scope” systems not easily virtualized, can be modeled orsimulated using virtual services so that a virtualized in-scope systembelieves it is talking to the live out-of-scope system modeled by thevirtual services. A virtual service can be used to bring all of theneeded systems into the cloud. In-scope systems can be provisioned withvirtual machines, and out-of-scope systems provisioned as virtualservices in order to bring an entire test lab into the cloud. In someinstances, capturing or generating a virtual service can includerecording live traffic between an in-scope system and its immediatedependencies. The recorded traffic can be used to create a model (orvirtual service) of the dependencies, such as described, for example, inU.S. patent application Ser. No. 12/242,783 to Michelsen (filed Sep. 30,2008). Generated virtual services can then stand in place for the livesystem and be used in the virtual testing of the system (anddependencies). As a result, in instances where live virtual machines areprovisioned for in-scope systems (e.g., a system under the user'scontrol), and virtual services are provisioned for the out-of-scopesystems (that would otherwise have required off-cloud connectivity),elastic capacity consumption and provisioning efficiency of a fullycloud-based testing environment can be realized.

Application servers 106, 108 can each include one or more processors140, 142, computer-readable memory 150, 152, and one or more interfaces.Application servers 106, 108 can include any suitable software componentor module, or computing device(s) capable of hosting and/or serving asoftware application (e.g., 112, 114), including distributed,enterprise, or cloud-based software applications. For instance,application servers can be configured to host, serve, or otherwisemanage web services or applications (e.g., 112, 114), such as SOA-basedor enterprise web services, or applications interfacing, coordinatingwith, or dependent on other enterprise services. Application andservices 112, 114 provided through application servers 106, 108 canfurther include web services under development. In some instances, somecombination of one or more of application servers 106, 108 can be hostedon a common computing system, server, or server pool, and sharecomputing resources, including shared memory, processors, andinterfaces, such as in an enterprise software system serving services toa plurality of distinct clients and customers.

The illustrated implementation of FIG. 1 further includes one or morelocal and/or remote clients 102, 104. A client 102, 104 can be anycomputing device operable to connect or communicate at least with anapplication server 106, 108, network 120, and/or virtual testingenvironment 110 (e.g., via test manager 170) using a wireline orwireless connection. Each client 102, 104 can include at least onegraphical display device and user interfaces (e.g., 160, 162), allowinga user to view and interact with graphical user interfaces of testmanagement software. Such graphical user interfaces can includeinterfaces for use in launching a test, providing instructions andparameters for a test, editing or otherwise modifying a test, viewingdetails and progress of a test, viewing health of the test system,viewing the provisioning of hardware resources for a test, etc. Ingeneral, the client 102, 104 can include any electronic computing deviceoperable to receive, transmit, process, and store any appropriate dataassociated with the software environment of FIG. 1. It will beunderstood that there may be any number of clients 102, 104 associatedwith system 100, as well as any number of clients 102, 104 external tosystem 100. Further, the term “client” and “user” may be usedinterchangeably as appropriate without departing from the scope of thisdisclosure. Moreover, while each client 102, 104 is described in termsof being used by one user, this disclosure contemplates that many usersmay use one computer or that one user may use multiple computers.

While FIG. 1 is described as containing or being associated with aplurality of elements, not all elements illustrated within system 100 ofFIG. 1 may be utilized in each alternative implementation of the presentdisclosure. Additionally, one or more of the elements described hereinmay be located external to system 100, while in other instances, certainelements may be included within or as a portion of one or more of theother described elements, as well as other elements not described in theillustrated implementation. Further, certain elements illustrated inFIG. 1 may be combined with other components, as well as used foralternative or additional purposes in addition to those purposesdescribed herein.

Traditionally, a common use case for cloud infrastructure wasin-production implementation of software systems. The elastic capacityand run time management capabilities of cloud-based systems can also beleveraged for in-development use cases, as well as test labs. Cloud canrealize efficiencies in instances where the volatility of demand variesamong a variety of uses of a particular infrastructure. For instance,different applications have different capacity needs over time. Theability to leverage one common resource pool among many teams, projects,and test can give the appearance of higher capacity on a per team orproject basis, when in reality, the unused capacity of other teams orprojects is being used. Further, during development, multiplepre-production, development, reduction, and test labs can be utilized inconnection with a single production infrastructure. Each of such labscan additionally have a volatile demand for capacity. Additionally,provisioning volatility can be particularly high within preproductiondevelopment labs, in some instances as high as in productioninfrastructure. Accordingly, cloud provisioning capabilities can beleveraged within pre-production use cases.

Leveraging cloud infrastructure for preproduction use can furtherinclude pooling the resources of several pre-production and testingteams that will be leveraging the infrastructure together. This caninclude establishing a single environment from the collective resourcesof several environments involved in a particular production effort.Accordingly, a virtual environment, or lab, provisioning system can beimplemented to dynamically provision pre-production needs for computingresources. In some instances, one or more catalogs can be maintained ofall the computing resources included within the single environment. Sucha catalog can further consist of virtual machine images of each of thesystems and resources the various pre-production teams and personnel mayneed at any given time during pre-production or testing. The virtual labprovisioning and management system can then allow administrators toleverage the components to realize pre-production goals, tests, andother tasks. Further, additional virtual machine images can be added tothe catalog, extending the scope of the provisioning system whilepotentially doing away with the need for additional physical hardwareabove and beyond that provided by the cloud.

In one example, one or more development or testing teams can have theircatalog of virtual machines provisioned dynamically, onto one or morevirtual machines. The provisioning of a set of images on virtualmachines can be referred to as a “virtual lab.” Such a lab can beprovided, for example, through a virtual lab management solution, as aself-contained unit that can be provisioned, decommissioned, and securedaltogether. Further, unlike some traditional testing environments thattake days or in some cases weeks to build and provision usingtraditional techniques, a virtual test lab can be set up using a virtuallab management solution, with the acquisition and installation ofhardware and base software reduced from weeks or months to, in somecases, minutes or seconds.

To further leverage the elastic capacity for performance testingcomputing systems and software products, a virtual lab managementsolution can be provided that is further capable of self-monitoring toensure that inefficiencies and errors in the virtual lab managementsolution do not influence or corrupt the returned test results. Suchself-monitoring can enable a performance testing tool to understand itsown load generation and resource consumption in order to then makepredictions with regard to how much, if any, additional resources mayneed to be allocated, from the cloud, to meet the performance testingrequirements for a particular test. Having such awareness, a testingtool can then go to the cloud and provision the additional resources onan effectively as-needed basis, and apply the configuration and testingassets of the test lab to those additional, dynamically-provisionedmachines, thereby seamlessly integrating them into the test lab as thetest is running. Such capabilities can be used to further leverage cloudinfrastructure for performance testing in an on-demand, utility typefashion.

Turning to FIG. 2, a schematic representation 200 is shown of a virtualtesting environment 110 performing a particular example virtual test. Inthis particular example, a user of a client workstation 205 can interactwith a test manager 210 to initiate a virtual test lab executed on a setof computing devices included in a server pool or cloud architecture. Aregistry 215, including one or more data stores, can be provided inconnection with the test manager 210, for instance, to receive, store,and manage test results returned from one or more tests. The testmanager 210 can be executed on a computing device communicativelycoupled to the client workstation 205, such as a device (e.g., 106)associated with development of a particular software system orcomponent. In other instances, test manager 210 can be executed locallyon the workstation 205 or remotely and provided to the user of theworkstation as a service. In any event, the test manager 210 can beexecuted on one or more persistent computing resources. In connectionwith the launching of a virtual test lab to test a particular softwaresystem or element, including systems and services executed asvirtualized hardware, systems, and services, a virtual test labcoordinator 220 can also be provisioned dynamically, together with oneor more test simulators 225, 230, 235. Additional simulators (e.g., 240)can be provisioned for the test, on demand and as needed, by allocatingand provisioning additional cloud-based computing devices with images ofthe test simulators used in the test.

In some instances, one or more users using workstation 205 can launch aparticular lab in the cloud simply by starting a testing activity. Insome instances, launching a particular test lab can include staging theentire test to run in the cloud. In such instances, there would be norequirement for using off-cloud hardware to perform the test. Theworkstation 205 can interface with test manager 210 in the launching ofa test. The test manager 210 can provision virtual-machine-baseddevelopment and test labs, and coordinate and launch correspondingvirtual service environments (VSEs) and automated regression andperformance test servers in the cloud for use during the test.

In some instances, provisioning and launching a test lab, using testmanager 210, can include utilizing elements and functionality includinga hypervisor to host machine images (e.g., using a Platform as a Service(PaaS) offering), a provisioning facility to manage and orchestrate thetest environment (e.g., using an Infrastructure as a Service (IaaS)offering), and service virtualization tools, such as iTKO's LISA™testing product, to solve for issues specific to cloud-based test labsas well as issues related to off-cloud, unavailable, costly, or highlydata-volatile systems that a test or team depends upon during testing(and/or development generally). In preferred embodiments, the test lab,including coordinator 220 and simulators 225, 230, 235, 240 can becompletely self-contained in the cloud. While live, off-cloud systemsand dependencies can be utilized during testing, such dependencies cannegate certain rapid provisioning and capacity benefits realized inpurely cloud-based labs. Virtualization of both the in-scope systemresources as virtual machines, and the capture and simulation ofoff-cloud or unavailable resources as virtual services, can allowdevelopers, testers and performance engineering teams to work inparallel at a fraction of the expected infrastructure cost.

In some instances, test manager 210 can access and identify, from acatalog, or other collection of pre-production resources (in some cases,from several different development teams leveraging the infrastructuretogether), a set of pre-production resources, and/or images of suchresources needed for a particular virtual test lab. The test manager 210can identify a particular pre-defined test or test lab module anddynamically provision one or more allocated computing devices with thetest lab resources, including images and simulators, corresponding tothe requested test or test lab on the cloud. Dynamic provisioning ofresources can include provisioning cloud-based resources with virtualassets (e.g., from the catalog) representing or virtualizing physical or“real world” computing resources needed for the test.

Each virtual test lab or test can include a corresponding testcoordinator 220 adapted to interface with the test manager 210, transmittest results to the test manager (e.g., for storage in registry 215), aswell as monitor performance of the test together with performance of thecloud-based computing devices used to implement the test lab. The testcoordinator 220 can monitor the progress of the test, such as bymonitoring how quickly a test is being completed, as well as bypredicting and managing subsequent test steps or flows. For instance, ina load testing test lab that will apply a varying load to the systemunder test, the test coordinator 220 can monitor progress of the testboth from a historical perspective (based on the portion of the testalready been completed) as well as predictively, anticipating subsequentsteps in the test. Continuing with the load testing example, the testcoordinator 220 can note performance status of a test during a firstportion of the test modeling a load of 10,000 virtual users and notefurther that a subsequent portion of the test will increase the load to50,000 virtual users. Consequently, the test coordinator 220 can gatherand analyze, continuously and in real time, statistical data generatedduring monitoring of the test and predict, using this statistical data,how expected subsequent test events or stages will affect performance ofthe test.

In addition to monitoring the status of its corresponding test, a testcoordinator 220 can monitor the cloud-based hardware implementingaspects of the test and test lab, including hardware implementingsimulators 225, 230, 235, 240. Simulators 225, 230, 235, 240 can modelvarious systems, users, operations, and events interacting with oraffecting the system under test. Continuing with the load test example,simulators 225, 230, 235, 240 can simulate client systems requesting andconsuming services provided by the system under test. Depending on thespecific requirements and steps of a particular test, the test labitself can be subject to varying capacity requirements during the courseof a test, as some test steps and operations (modeled by simulators 225,230, 235, 240) can be more resource-intensive that others. In order tomitigate against the test lab under- or incorrectly-performing, the testcoordinator 220 can monitor the cloud hardware implementing thesimulators 225, 230, 235, 240 to ensure that sufficient resources havebeen allocated to successfully and efficiently complete the test (or asuite of tests). For instance, the test coordinator 220 can monitor the“health” of each of the cloud-based (or server-pool-based) computingdevices initially allocated for the test and determine whether thedevices have been over- or under-allocated to the needs of the testgenerally or during a certain period in the test. Determining the healthof the computing devices can include monitoring network (e.g.,bandwidth) capacity and performance, processor (e.g., CPU) capacity andperformance, memory (e.g., heap) capacity, as well as the respectiveloads (including network, processing, and memory loads) borne by each ofthe computing devices during the test. In some instances, a compositehealth score can be generated for a computing device from a plurality offactors. The health (and load) of a particular computing deviceallocated for a particular test can depend on a number of factors,including the particular task(s) or simulator(s) executed using thedevice, the individual hardware and network characteristics of theparticular device, and the extent to which, if any, the particularcomputing device is being used for other, parallel tasks.

If the test coordinator 220 determines that the set of computing devicesused in the test is incorrectly (i.e., over- or under-) allocated, thetest coordinator 220 can dynamically provision additional computingdevices with additional simulators, for instance, or tear-downpreviously-used computing devices, in accordance with the determination.Additionally, the test coordinator 220 can manage the allocation ofcomputing devices predictively. As an example, the test coordinator 220need not wait for one or more computing devices used in the test toenter a critical health state or begin underperforming before initiatingthe provisioning of additional computing devices. For instance, the testcoordinator can begin collecting statistical data during the beginningmoments or periods of a test and make predictions, based also on theexpected progression of the test, whether the current allocation ofcomputing devices is likely to be sufficient or optimal over the short-or long-term duration of the test. Accordingly, the test coordinator 220can proactively allocate additional computing devices (or de-allocatecomputing devices) in the set based on statistical data collected duringearlier phases of the test.

Turning to FIG. 3, a flowchart 300 is shown illustrating an exampletechnique, performed, for example, by test coordinator 220 and/or testmanager 210, in connection with the dynamic provisioning of computingdevices in connection with an example virtual test. At least one virtualtest can be launched 305 to test aspects, components, hardware, module,subsystems, etc. of a computing system under test. The test can utilizea second computing system, such as a cloud-based service or test serverpool that includes a plurality of computing devices that can beallocated for the test on-demand. Progress of the test can be monitored310 during a first period of time. Performance of the second computingsystem can itself be monitored 315 during the first period. Based atleast in part on the monitoring of the test progress and monitoring ofthe performance of the computing system during the first time period, anadditional set of computing devices can be dynamically provisioned 320for inclusion in the second computing system for use in continuingperformance of the test during a second period of time subsequent to thefirst period.

Turning to FIGS. 4A-4E, simplified block diagrams 400 a-e are shownillustrating some potential operations associated with the presentdisclosure. In the examples of FIGS. 4A-4E, an example load test isperformed using virtual testing systems and according to principlessimilar to those described above. In some instances, virtual loadperformance testing can be well-suited to a cloud-based or on-demandhardware provisioning environment. In the implementation of FIG. 4A, avirtual test lab 405 includes a test coordinator tool 410 and one ormore initial simulators 415 simulating load on the system under test. Inthis example, and for simplicity, each of test coordinator 410 andsimulator 415 are executed on distinct computing devices allocated andprovisioned for the test from a plurality of available computingdevices. Test coordinator 410 can be responsible for managing aparticular load performance test according to load profile 420 thatinitially ramps up to a first peak load 425, drops to a first trough430, then ramping back up to a second peak 435 before concluding (e.g.,at time t=10).

Further, the health or performance of the hardware implementing the testcan be monitored in whole or in part (e.g., in cooperation with apersistent test manager tool) by the test coordinator 410. As shown inFIG. 4A, represented by the graph 440, the health of the computingdevice implementing the initial simulator 415 is monitored. The healthof the computing device can be based on the capacity of the device,including its remaining bandwidth, memory, and processing capacity. Inthe example of FIG. 4A, when the test first begins, an initial period ofthe test's execution can be monitored to develop a statisticallymeaningful set of test and hardware performance data, upon which futurehardware assessment decisions and forecasts can be based. Accordingly,performance of the test and test hardware can be continuously monitoredby the test coordinator 410 in order to develop a dataset documentingperformance of the test lab. Periodically, the test coordinator 410 candetermine that statistically sufficient data samples have been collectedto justify forecasting, calculating, or otherwise determining whetheradditional hardware should be allocated and provisioned, dynamicallyduring the test, to meet the needs of the test. This allows for the testcoordinator 410 to calculate, and adjust, in real time the minimum, oroptimal, amount of hardware resources needed for a particular test ortest period.

The test coordinator 410 can periodically assess whether the testhardware allocation is sufficient or optimal based on collectedstatistical data. Such assessments or the rate of such assessments canbe pre-scheduled, pre-defined, or based on pre-defined rules andconditions. For instance, a user initiating the test may be aware ofsections of a test that are particularly important or that the userpredicts are likely to tax the test lab hardware. Accordingly, the usercan predefine checkpoints for assessing the sufficiency of the test labhardware. For instance, a user can specify that assessments be made morefrequently between times t=2 through t=4 (as shown at 420) given anexpectation that the capacity of the test lab hardware is most likely tobe challenged during this period.

In addition to specifying checkpoints and assessment frequency, asdiscussed above, users can also pre-define limits for hardwareallocation. Users can specify such parameters, for instance, inconnection with the initiation of a test (e.g., using workstation 205).Setting a maximum limit on the amount of hardware that can bedynamically allocated by the test coordinator 410 during the test can beuseful, for instance, where testing hardware resources are allocatedthrough a cloud computing service that bills users according to thenumber of computing resources allocated per unit of time. While it maybe ideal to ramp up the number of dynamically provisioned hardware to acertain number of servers within a particular performance test, such atest may provision hardware at a cost in excess of what theuser-customer is in a position to pay. Accordingly, some control can bemaintained by the user to artificially restrain the dynamic provisioningof hardware during a test, in order to address certain costrequirements. In still other examples, users can set a conditionalmaximum hardware allocation, specifying that the maximum may only beexceeded under certain circumstances (e.g., where adhering to themaximum would cause a test to fail, where the test is near completion,where exceeding the maximum allocation is determined to only betemporary, etc.). Further, a user can also set an initial hardwareallocation for a test, in connection with launching of a test. Indeed, auser can specify a number of test parameters including the initialserver outlay, server outlay limits and maximums, desired assessmentpoints, rates, and protocols, among other parameters.

In some instances, a test (e.g., 420) can be designed to allow for awarm-up or ramp-up period at the beginning of the test, or test section,to assist the test coordinator 410 in gathering test performancestatistics for a period before dynamic provisioning of additionalhardware would be needed. It can be desirable to postpone decisionsregarding the dynamic allocation or de-allocation of test server untilat least a statistically reliable sample is collected, such as to avoidinstances where a small sample of outlier data is initially gathered andthen relied upon to incorrectly or prematurely provision additionalhardware, or worse, de-allocate needed hardware. Further, were a test toramp-up too quickly, in terms of its hardware resource requirements, itis possible that the health of the provisioned test hardware would reacha critical or overloaded state before statistically significant data canbe collected and acted upon to stave off a critical device event. Inaddition to ramp-up periods built into a test flow, the test coordinator410 can further determine or predict a minimum amount of hardware thatshould be allocated to the test to allow the test coordinator 410 to atleast successfully complete the warm-up period (e.g. in connection withan initial server outlay specified during launching of the test).

In FIG. 4A, the performance of simulator (415) hardware can be monitored(at 440) during a ramp-up state, the capacity approaching 20% at timet=2. In these examples, “capacity” is used generically for simplicity,but in practice, can apply to the bandwidth, memory capacity, processorcapacity, combined health of two or more of bandwidth, memory capacity,and processor capacity, among other considerations and measures. In theexample of FIG. 4A, from the ramp-up period, it may appear to the testcoordinator tool 410 that sufficient computing resources have beenallocated for the load test. However, turning to FIG. 4B, it can bedetermined or forecast, that by time t=4 (for instance, in connectionwith load peak 425), the capacity of the resources is at or trending toan unacceptable level. Various thresholds can be applied in determiningwhether a device's capacity or capacity trend is unacceptable at anygiven moment in any particular test. For instance, exceeding 100% of anallocated device's memory, processing, or networking capacity can causethe device and/or the test to fail, making it unacceptable to even flirtwith exceeding devices' capacity. Accordingly, in some examples, thedynamical provisioning of additional devices can be prompted well inadvance of a device meeting or exceeding its capacity. This can alsohelp account for delays in computing or responding to a device reachinga critical state. For instance, in one example, a device may bedetermined to be successfully running at 80% capacity, based onstatistical data collected during test monitoring that suggestsoperations and simulations handled by the device, on average, will notcause the device to fluctuate unpredictably from 80% to beyond 100%.This notwithstanding, the possibility may still exist, based on one ormore outlying data points gathered during performance of the test, thata particular event or operation could cause a spike in load on thedevice, sending the device into an unacceptable state. Accordingly,thresholds for initiating dynamic provisioning of additional devices canthemselves fluctuate dynamically, based on statistical data gatheredduring earlier stages of a test.

Based on the forecast shown at 440 in FIG. 4B, the test coordinator 410can elect to allocate and provision additional hardware resources tomirror and/or supplement simulator 415. For instance, in the example ofFIG. 4C, two additional simulators (415 b and 415 c) and computingdevices have been provisioned in addition to the originally provisionedsimulator 415 a, in response to predicting that using the singlesimulator (415 in FIGS. 4A and 4B) would be insufficient or less thanideal. The test coordinator 410 orchestrates the allocation of theseadditional devices together with the provisioning of the allocateddevices with the resources, virtual machine images, etc. needed toseamlessly integrate the additional computing power and simulators inthe test. The test coordinator 410 performs this allocating andprovisioning (for simplicity, also referred to collectively as“provisioning”) dynamically during the test and without the interventionof a human user. Additionally, in this particular example, the testcoordinator 410 has preemptively provisioned these additional computingdevices based on a prediction that the trend of the test indicated thatthe additional computing devices should be provisioned. The effect ofthis provisioning can be seen, as illustrated at 440 a-440 c in FIG. 4C,in the respective health measures of each of the three provisioneddevices, each trending at a rate that appears to allow for the test tocomplete without incident.

The dynamic provisioning of additional testing hardware during a testcan be adjusted according to the needs and progress of a test. From astatistical standpoint, the longer the test is monitored, the more datapoints will have been gathered by the test coordinator 410 for use inmaking predictions concerning performance trends and the adequacy ofprovisioned hardware. Accordingly, a previous decision to provisionadditional hardware can be amended based on additional data, orsubsequent test phases, to add more test servers or de-allocate andtear-down previously-used or -added test servers later determined to beunnecessary, such as shown in the example of FIG. 4D. As shown in thehealth monitoring graphs 440 a-c, hardware resources are (or arepredicted to be) underutilized after time t=5. Accordingly, anassessment can be made to de-allocate one of the simulator's devices 415c, as shown in FIG. 4E, based on a prediction that the two simulators415 a, 415 b, and corresponding hardware, will be sufficient to completethe test (as shown in health monitoring graphs 440 a-b).

Further, while some of the non-limiting examples described above discussa single, cloud-based “test coordinator” that both gathered data, madehardware assessments based on that data, and provisioned additionalresources based on the assessments, other implementations can make useof a persistent test manager in combination with the test coordinator.Indeed, all or a part of the functionality described above can beembodied in one or both of a dynamic (e.g., cloud-provisioned) testcoordinator and a persistent test manager. For instance, in one exampleimplementation, a cloud-based test coordinator can monitor and collectdata from related cloud-based hardware and transfer this data forfurther analysis to a persistent test manager. The test manager can acton this data by requesting the dynamic provisioning of additionalhardware in connection with the test. The test manager can then requestthe assistance of the test coordinator in provisioning additionalhardware for a test, based on the test manager's assessment.Additionally, and by way of another non-limiting example, the dynamicprovisioning of additional hardware for simulators for a particular testcan be the domain of the test coordinator, while the dynamicprovisioning of additional test coordinators, corresponding to newtests, can be the domain of the test manager (such as with the parallelexecution of a suite of test as discussed, for example, below). Otherimplementations, including implementations utilizing other components orother allocations of functionality between persistent and dynamiccomputing resources are also within the scope of this disclosure.

While the examples of FIGS. 4A-4E concern execution and monitoring of asingle virtual test, in some instances, a single user or organizationmay wish to complete multiple tests or a suite of tests within a singletest session. For instance, a user may request that a plurality of testbe completed within a given timeframe. Indeed, implementations makinguse of cloud infrastructure can also be leveraged to support multipleconcurrent tests, including tests of virtualized systems and testsinvolving the use of virtual services.

As discussed above, in some implementations, a single test coordinator(e.g., 220 or 410) can be provisioned for each test launched, forexample, using a test manager (e.g., 210). Test coordinators can collectdata from tests and hardware running the tests and forward this data toa test manager, among other functionality. Turning to the examples ofFIGS. 5A-5D, a test environment 505 includes a plurality of test labseach including a test coordinator (e.g., 510, 515) and one or moresimulators (e.g., 520, 525). The test environment can allocate andprovision a plurality of available test servers for use in executingtests in the test suite. In the specific example illustrated in FIG. 5A,a user has requested that a suite of 50 tests (represented by the y-axisof graph 530 a) be executed within a particular time period of 6 hours(represented by the x-axis of graph 530 a). As a theoretical matter, inorder to complete these tests within the time allotted, it can beinitially calculated (e.g., using a test manager (e.g., 210) or one ormore test coordinators (e.g., 220)) that tests in the suite be completedat a certain average rate 535. Further to achieve this goal, it isdetermined, in this particular example, that at least two tests willneed to be executed concurrently within test environment 505 at any onetime. Accordingly, to achieve this goal, the test environment has beeninitially provisioned with two coordinators 510, 515 corresponding withtwo concurrently-executed tests, each having its own set of simulators(520, 525, respectively) responsible for simulating conditions andtransactions in the test. In the example of FIG. 5A, more than one testcoordinator is executed by a single allocated computing device 508. Inother examples, each test coordinator can be executed on a dedicatedcomputing device.

As execution of the suite of tests commences, the initial assumptionsregarding how many tests will need to run concurrently can change. Forinstance, some tests may be executed faster than predicted, while otherstake longer to execute. As in the examples of FIGS. 4A-4E, asstatistical data is collected from the collection of tests beingexecuted within a given session, more reliable predictions can be made,given the ever increasing collection of data points that are gatheredduring monitoring of the tests' performance. For instance, turning tothe example of FIG. 5B, at time t=1.5 hours, only six tests have beencompleted (or are forecast to have been completed). Further, it can bepredicted, based on the statistical data collected over theinitially-executed tests, that continuing with the execution of only twoconcurrent tests at a time will not realize the goal of completing all50 tests in 6 hours, as indicated by the actual trend line 538 a, asforecast at t=1.5. Accordingly, it can be determined, using a testmanager, a centralized test coordinator collecting data from each of thetest suite's respective test coordinators and managing execution of thesuite of tests in a session, that additional test labs should beconcurrently launched, in order to keep pace with the session goal. Inother words, statistical data collected from the execution of a firstportion of the suite of tests can indicate that the mean execution timeof the tests was higher than originally predicted, leading to asuggested upgrade in the number of hardware that should be provisionedfor the suite of tests.

In the example of FIG. 5B, it has been determined that four tests willneed to concurrently execute in order to reach the six hour goal.Accordingly, two additional test coordinators (e.g., 540, 550) can beprovisioned, along with provisioning of additional hardware for thecorresponding test simulator sets 545, 555. While a single computingdevice 508 was originally provisioned for the testing session for thesuite of tests, the originally-provisioned device(s) (e.g., 508) may notbe have the capacity to successfully execute additional testcoordinators (corresponding to additional, concurrently executed tests).For instance, as shown in FIG. 5B, the doubling of test coordinatorsexecuted on device 508 has resulted in an untenable health profile trend560 a for the device 508. Accordingly, as shown in FIG. 5C, the additionof additional, concurrently-executed tests can necessitate the dynamicprovisioning of additional computing devices (e.g., 565), for instance,from the cloud.

As shown in FIG. 5C, by executing test coordinators 512, 518, 540, 550on more than one provisioned computing device (e.g., devices 508 and565), device health can be maintained or predicted over the life of thetest (as shown at 560 b and 570 a). Further, as shown by trend line 538b, using the two provisioned computing devices 508, 565 in connectionwith the execution of at least four concurrent tests (and correspondingtest coordinator tools), the testing of the suite appears to be set tomeet the originally defined six hour goal. Assumptions and trends canchange, however, as additional samples and data are collected frommonitoring of the tests' progress, even after the provisioning ofadditional computing devices (e.g., 565) that appear to fulfill theneeds of the test. For instance, as shown in FIG. 5D, at t=4.5 hours, atrend (shown at 538 c) can be identified that requires the addition ofstill more computing devices. Accordingly, in the example of FIG. 5D, inorder to address trend 538 c, the execution of an additionalconcurrently-executed test is added, along with the provisioning of oneor more additional computing devices (e.g., 570) to accommodate thecorresponding, additional test lab (and test coordinator 575). In thisway, a testing system can monitor itself and intelligently self-correct,in an at least partially iterative manner, predicted shortfalls (orexcesses) in provisioned computing resources.

In some instances, rather than determining that a particular number oftests be concurrently executed at a given time throughout a testsession, a particular number of computing devices can be allocated tohandle the execution of a set of concurrent tests. In principle, thisproceeds in a manner similar to the examples illustrated in FIGS. 5A-5D,with the concurrent testing being constrained by the number of teststhat can potentially be run on a particular allocated set of computingdevices, rather than artificially constrained by a pre-set number oftests that are to be concurrently executed. The allocated set ofcomputing devices can be provisioned to execute any combination ofsimulators and coordinators needed during the concurrent execution ofthe tests. Further, the test manager can determine when tests in thesuite are to run, so as to maximize or optimize the available computingdevices allocated for the testing session. For example, if a first testis being run using 80% of the available allocated test servers, and asecond concurrent test has just ended that used 10% of the remainingallocated test servers, the test manager tool can attempt to identifyanother test that the test manager tool predicts will also utilizeapproximately 10% of the allocated resources. Further, as assumptionsconcerning the test outlay change (e.g., if testing falls behind a paceneeded to complete a suite of tests), additional test servers can beallocated, as in the preceding examples, both to accommodate concurrenttesting of certain combinations of tests that unexpectedly overwhelm theprovisioned computing resources, as well as to accommodate meetingparticular testing time deadlines, or test completion rates.

In still additional examples, it should be appreciated that principlesof the examples of FIGS. 4A-4E (focused on the monitoring and dynamicprovisioning of hardware for a single test) and the examples of FIGS.5A-5D (focused on monitoring and dynamic provisioning in order toaccommodate concurrent testing) can be combined. For instance, whilecomputing devices are monitored and dynamically provisioned toaccommodate additional tests and test coordinators for concurrentexecution of a suite of tests (as in FIGS. 5B-5C), individual tests inthe suite, through their individual test coordinator tools, can, at thesame time, be initiating the dynamic provisioning of additional testservers (e.g., to add test simulators for the test) in connection withattempting to realize successful completion of the individual tests (asoutlined in connection with the example of FIGS. 4A-4E). For instance,after test server 565 is added (in FIG. 5C) to accommodate testcoordinator 540, an initial set of test servers provisioned forcorresponding simulators 545 can monitored by test coordinator 540 andthen expanded by dynamically provisioning additional computing devicesfor executing additional simulators 545. Indeed, in some instances, inorder to hasten execution of a particular suite of tests, test serverscan be added not only to facilitate provisioning of additional testcoordinator tools for additional tests (as in FIGS. 5A-5D), butadditional test servers can also be allocated and provisioned withadditional simulators, upon predicting that adding additional simulatorsto a particular test in the suite is likely to speed-up execution of theparticular test, and thereby also the completion of the entire testsuite.

It is important to note that the examples illustrated and described inconnection with FIGS. 4A-4E and 5A-5D are simplified examples, presentedfor convenience in introducing and describing particular concepts.Indeed, the real world test flows, threshold values and algorithms usedto trigger dynamic provisioning of test hardware, load patterns, healthmonitoring data, trend lines, test labs, monitoring techniques, andsimulator outlays, can be far more complex than (but, in some cases, assimple as) the examples and systems described in connection with FIGS.4A-4E and/or 5A-5D. As but one example, FIG. 6 shows an exampleperformance graph outlining monitoring of a particular real-world loadperformance test, including monitoring of the load pattern 610 testedagainst a computing system under test, as well as the number oftransactions per second 605 executed by one or more provisionedcomputing devices used in the execution of the test. Other examples caninclude plots and representations of other measures and statistics.Further, it is also important to note that potentially limitless testtypes, simulator types, test plans, etc. can be accommodated usingvirtual testing systems implementing principles similar to thosedescribed above.

By way of additional illustration, and in one particular example, avirtual testing system with functionality similar to that describedabove, can be accessed, controlled, and/or monitored, by human users byway of computing devices (e.g., 102, 104) that include graphical displaycapabilities. To illustrate certain simplified features of oneimplementation of an example virtual testing system, FIGS. 7A-7C showcertain simplified representations of screenshots and views 700 a-c thatcould be displayed to users in connection with the set-up and executionof a test in a virtual test lab dynamically provisioned using a virtualtesting system. For instance, FIG. 7A illustrates a view 700 a of a userinterface for use, by a user, in launching one or more virtual testlabs. The user can navigate a collection of test labs using window 705and select one or more particular test labs (e.g., 708) included in thecollection. By pressing the “Start Lab” button 710 in the interface, theselected test lab can be initiated, for instance, by dynamicallyrequesting the allocation of a set of on-demand servers and dynamicallyprovisioning the allocated servers with the resources of the selectedtest lab. Using such an interface, the provisioning of a virtual testingenvironment can be initiated and completed swiftly and simply, with thetest manager tool performing the remaining transactions and operationsto set-up the test lab on the cloud. In some instances, selecting the“Start Lab” button 710 can conveniently result in the full provisioningand initiation of a test lab without further input from the user.

Turning to FIG. 7B, a view 700 b of another example user interface isshown. In the example of FIG. 7B, a schematic representation of aprovisioned test is presented to a user in window 715. Icons 720-745 canbe included and displayed, representing persistent and dynamiccomponents of the test environment participating in the launch of thetest. For instance, in the example of FIG. 7B, a test has been launchedby a test manager 720 making use of a particular development test lab725 testing one or more components of a particular computing system. Theuser interface further presents the components included in the test lab725 itself, including a test coordinator 730 provisioned on cloudinfrastructure, an initial set of one or more test simulator 735provisioned on cloud infrastructure, and a virtual service environment(VSE) 740 virtualizing the system under test (also provisioned on thecloud). Through the use of virtual services made available through VSE740, the test simulators 735 can interact with and test a virtualizedversion of the system under test, provisioned on the cloud as one ormore virtual services, rather than testing the live system itself.Additionally, the VSE 740 can provide virtual services virtualizingdependencies of the system under test.

As the test progresses, the test manager 720 and/or test coordinator 730monitors the test's progress, as well as the performance of theprovisioned devices implementing the test, to determine if additionaldevices should be dynamically provisioned. To assist the user inunderstanding the progress of the test and the automatic provisioning(or de-allocation) of any additional test servers, additional data canbe presented to the user via the user interface in FIG. 7B. For example,as shown in the example of FIG. 7C, additional simulators 760, 765, 770,775 have been dynamically provisioned on additional correspondingdevices, for instance, in response to determining that additionalsimulators were needed to complete the test. Additionally, test serverhealth status can be conveyed to the user graphically via windows780-790. Each status window 780-790 corresponds to a particular one ofsix computing devices provisioned by the system for the test coordinator730 and simulators 735, 760, 765, 770, 775. The status windows 780-790can display and synthesize data documenting or representing the device'shealth and capacity as monitored, for example, by the test manager 720and/or test coordinator 730 during the course of the test. For instance,in the example of FIG. 7C, windows 780-790 convey information includingeach provisioned server's respective memory (“heap”), CPU capacity(“cpu”), as measured against the average load (“avg load”) borne by therespective test server over time. With the information presented inwindow 715, showing the addition (and/or subtraction) of provisionedservers (and simulators, test coordinators, etc.), together with theinformation presented in windows 780-790, users can monitor how thevirtual testing system is responding to the dynamically-changinghardware requirements of the test. Such information can be further usedto inform users monitoring the interfaces of FIGS. 7B and 7C of how tobuild better tests in the future, adjust assumptions and instructionsfor a test (such as a maximum number of servers to be allocated to thetest or the assessment rate for the test), and can even make adjustmentsto the test, during testing (such as adding or changing assessmentcheckpoints), to reflect knowledge gleaned from the test's performance.

Although this disclosure has been described in terms of certainimplementations and generally associated methods, alterations andpermutations of these implementations and methods will be apparent tothose skilled in the art. For example, the actions described herein canbe performed in a different order than as described and still achievethe desirable results. As one example, the processes depicted in theaccompanying figures do not necessarily require the particular ordershown, or sequential order, to achieve the desired results. In certainimplementations, multitasking and parallel processing may beadvantageous. Additionally, other user interface layouts andfunctionality can be supported. Other variations are within the scope ofthe following claims.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on an artificiallygenerated propagated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal that is generated to encodeinformation for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices), including a distributed software environment orcloud computing environment.

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources. The terms “data processing apparatus,” “processor,” “processingdevice,” and “computing device” can encompass all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includegeneral or special purpose logic circuitry, e.g., a central processingunit (CPU), a blade, an application specific integrated circuit (ASIC),or a field-programmable gate array (FPGA), among other suitable options.While some processors and computing devices have been described and/orillustrated as a single processor, multiple processors may be usedaccording to the particular needs of the associated server. Referencesto a single processor are meant to include multiple processors whereapplicable. Generally, the processor executes instructions andmanipulates data to perform certain operations. An apparatus can alsoinclude, in addition to hardware, code that creates an executionenvironment for the computer program in question, e.g., code thatconstitutes processor firmware, a protocol stack, a database managementsystem, an operating system, a cross-platform runtime environment, avirtual machine, or a combination of one or more of them. The apparatusand execution environment can realize various different computing modelinfrastructures, such as web services, distributed computing and gridcomputing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, module, (software) tools, (software) engines, orcode) can be written in any form of programming language, includingcompiled or interpreted languages, declarative or procedural languages,and it can be deployed in any form, including as a standalone program oras a module, component, subroutine, object, or other unit suitable foruse in a computing environment. For instance, a computer program mayinclude computer-readable instructions, firmware, wired or programmedhardware, or any combination thereof on a tangible medium operable whenexecuted to perform at least the processes and operations describedherein. A computer program may, but need not, correspond to a file in afile system. A program can be stored in a portion of a file that holdsother programs or data (e.g., one or more scripts stored in a markuplanguage document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

Programs can be implemented as individual modules that implement thevarious features and functionality through various objects, methods, orother processes, or may instead include a number of sub-modules, thirdparty services, components, libraries, and such, as appropriate.Conversely, the features and functionality of various components can becombined into single components as appropriate. In certain cases,programs and software systems (e.g., system 100) may be implemented as acomposite hosted application. For example, portions of the compositeapplication may be implemented as Enterprise Java Beans (EJBs) ordesign-time components may have the ability to generate run-timeimplementations into different platforms, such as J2EE (Java 2 Platform,Enterprise Edition), ABAP (Advanced Business Application Programming)objects, or Microsoft's .NET, among others. Additionally, applicationsmay represent web-based applications accessed and executed via a network(e.g., through the Internet). Further, one or more processes associatedwith a particular hosted application or service may be stored,referenced, or executed remotely. For example, a portion of a particularhosted application or service may be a web service associated with theapplication that is remotely called, while another portion of the hostedapplication may be an interface object or agent bundled for processingat a remote client. Moreover, any or all of the hosted applications andsoftware service may be a child or sub-module of another software moduleor enterprise application (not illustrated) without departing from thescope of this disclosure. Still further, portions of a hostedapplication can be executed by a user working directly at a serverhosting the application, as well as remotely at a client.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), tablet computer, a mobile audio or videoplayer, a game console, a Global Positioning System (GPS) receiver, or aportable storage device (e.g., a universal serial bus (USB) flashdrive), to name just a few. Devices suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device, includingremote devices, that are used by the user.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include any internal or external network,networks, sub-network, or combination thereof operable to facilitatecommunications between various computing components in a system (e.g.,100). A network may communicate, for example, Internet Protocol (IP)packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells,voice, video, data, and other suitable information between networkaddresses. The network may also include one or more local area networks(LANs), radio access networks (RANs), metropolitan area networks (MANs),wide area networks (WANs), all or a portion of the Internet,peer-to-peer networks (e.g., ad hoc peer-to-peer networks), and/or anyother communication system or systems at one or more locations.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults.

What is claimed is:
 1. A computer-implemented method comprising:launching at least one test of a first computing system utilizing asecond computing system including a first set of computing devices;monitoring progress of the test during a first period of time;monitoring performance of the second computing system during the firstperiod; automatically provisioning an additional, second set ofcomputing devices for inclusion in the second computing system based atleast in part on the monitoring of the test progress and monitoring ofthe performance of the computing system during the first time period;and wherein the test utilizes the first and second sets of computingdevices during a second period of time subsequent to the first period.2. The method of claim 1, wherein the first and second sets of computingdevices are remote from the first computing system.
 3. The method ofclaim 1, wherein the second computing system executes at least oneinstance of a virtual test lab configured to simulate a set ofinteractions with the first computing system, and the virtual test labis executed on at least one of the first set of computing devices andautomatically provisioning the second set of computing devices includesreplicating instances of the virtual test lab on at least some of thesecond set of computing devices.
 4. The method of claim 3, wherein eachinstance of the virtual test lab is executed using at least one firstvirtual machine and a virtual instance of the first computing system istested, the virtual instance of the first computing system executedusing at least one second virtual machine.
 5. The method of claim 1,wherein monitoring the test during the first time period includespredicting requirements of the test in the second period of time and thesecond set of computing devices are provisioned based, at least in part,on the predicted requirements of the test.
 6. The method of claim 1,wherein monitoring performance of the second computing system includesmonitoring computing capacity of the first set of computing devices. 7.The method of claim 6, wherein computing capacity includes at least oneof processing capacity of computing devices in the second computingsystem, available network bandwidth within the second computing system,and available memory of computing devices in the second computingsystem.
 8. The method of claim 1, wherein the first and second sets ofcomputing devices are provisioned from a plurality of cloud-based serverdevices.
 9. The method of claim 1, further comprising: monitoringperformance of the second computing system during the second period, thesecond computing system including the first and second sets of computingdevices; and determining, based at least in part on monitoringperformance of the second computing system during the second period,whether additional computing devices should be added to the secondcomputing system in a third period subsequent to the second period. 10.The method of claim 9, further comprising determining, based on themonitoring of the second computing system during the second period, thatat least a portion of the computing devices provisioned to the secondcomputing system be at least temporarily de-allocated from the secondcomputing system.
 11. The method of claim 1, wherein at least oneparticular computing device, remote from the first computing system andwithin the second computing system, executes at least one testcoordinator engine managing the test.
 12. The method of claim 11,wherein the at least one test coordinator engine monitors the test,monitors performance of the second computing system, and initiatesdynamic provisioning of additional computing devices for inclusionwithin the second computing system during the test.
 13. The method ofclaim 1, further comprising identifying user-defined parameters for thetest, wherein dynamic provisioning of additional computing devices forinclusion within the second computing system during the test is based atleast in part on the parameters.
 14. The method of claim 13, wherein theparameters include at least one of: a monitoring rate defining whenperformance of the second computing system should be monitored inconnection with a decisions to initiate dynamic provisioning of additioncomputing devices for inclusion within the second computing systemduring the test; an initial system size for the second computing system;and a maximum system size for the second computing system, wherein thesecond computing system will not be automatically provisioned withadditional computing devices in excess of the maximum system size. 15.The method of claim 1, wherein the at least one test includes aplurality of tests, the plurality of tests including a first test and asecond test executed at least partially in parallel, wherein each testin the plurality of tests has a corresponding test coordinator engineexecuted on the second computing system.
 16. The method of claim 15,wherein monitoring of the at least one test includes determining whetheradditional computing devices should be automatically provisioned toaccommodate execution of test coordinator engines for at least the firstand second tests.
 17. The method of claim 15, further comprising:identifying user-defined parameters for the at least one test, theparameters including instructions for completing each of the pluralityof tests within a particular time period, the particular time periodincluding at least the first and second periods of time, wherein theperformance of the second computing system during the first period ismonitored to determine whether additional computing devices should beautomatically provisioned for inclusion in the second computing systemin order to the plurality of tests within the particular time period.18. The method of claim 1, wherein performance of the second computingsystem is monitored substantially continuously during the test anddecisions to automatically provision the second computing system withadditional computing devices are made substantially periodically.
 19. Anarticle comprising a non-transitory, machine-readable storage devicestoring instructions operable to cause at least one processor to performoperations comprising: launching at least one test of a first computingsystem utilizing a second computing system including a first set ofcomputing devices; monitoring progress of the test during a first periodof time; monitoring performance of the second computing system duringthe first period; automatically provisioning an additional, second setof computing devices for inclusion in the second computing system basedat least in part on the monitoring of the test progress and monitoringof the performance of the computing system during the first time period;and wherein the test utilizes the first and second sets of computingdevices during a second period of time subsequent to the first period.20. A system comprising: a memory element storing data; a processoroperable to execute instructions associated with the stored data; and atleast one test management engine configured to: launch at least one testof a first computing system utilizing a second computing systemincluding a first set of computing devices; monitor progress of the testduring a first period of time; monitor performance of the secondcomputing system during the first period; and automatically provision anadditional, second set of computing devices for inclusion in the secondcomputing system based at least in part on the monitoring of the testprogress and monitoring of the performance of the computing systemduring the first time period; and wherein the test utilizes the firstand second sets of computing devices during a second period of timesubsequent to the first period.